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[57] ABSTRACT 

A system and device for, and method of, presenting multi- 
media information. In a client-server context, the invention 
includes a client that receives units of the multimedia 
information and presents the information on a presentation 
device. Each unit of information has an importance value 
assigned to it, which in an exemplary embodiment is indica- 
tive of the unit*s importance in relation to the quality of the 
presentation. The invention includes a mechanism for char- 
acterizing the performance capabilities of the system. For 
example, several conventional statistics may be gathered 
and analyzed concurrently with the streaming operation and 
before it begins. The invention includes a mechanism for 
inferring network conditions from the characterized perfor- 
mance. The server may then stream the units of multimedia 
information to the client at a streaming rate and adapt the 
streaming rate of the streaming in response to the impor- 
tance information and in response to the inferred network 
conditions. 
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SYSTEM, DEVICE, AND METHOD FOR relationships, in short, that the stream itself is coherent. 

STREAlVnNG A MULTIMEDIA FILE Inter-stream synchronism means that multiple related 

streams are presented in synchronism with respect to each 
CROSS-REFERENCE TO RELATED other. Concerning intra-stneam synchronism, all streams, for 

APPLICATIONS 5 the most part, should present data in order. However, users 

This application is related to the following U.S. patent ^^^e forgiving if some streams, such as video, leave out 

applications, all of which are assigned to the assignee of this ^^^^"^ P^^^^^s of the data than they ate of other streams, 
application and aU of which are incorporated by reference ^uch as audio, domg the same. A video stream with missmg 

data may appear a little choppy, but an audio stream with 
Device, System And Method Of Real-Time Multimedia '° ^f'^f ""^^ completely unintelli^ble^ CoDcerting 

Stre^ (Serial No. 08/636.417, to Qio-Fan Zhu, ■"•^'t^*"' 'y^'^^l'^'^'^^^' ^°''^] ^'^y »^ 

rr TT~^ V /v/^ , X , u , poor "hp synch " makmg the presentation appear and sound 

Manickam K. ^ ^^^^^ ^^^^ 

Sridhar, and M. Vedat Eyuboglu, filed Apr., 23, 1996); and network-based context, one simple model of pro- 
Improved Video Encoding System And Method Sen No. 15 ducing the information involves the consuming entity to 
08/711,702, to Manickam R. Sridhar and Feng Chi request the downloading of the multimedia information for 
Wang, filed on even date herewith.). an entire presentation from a server, storing the multimedia 
BACKGROUND information. Once downloaded, the client may then 

consume, or present, the information. Although relatively 

1. Field of the Invention 20 simple to implement, this model has the disadvantage of 
The invention generally relates to real-time multimedia requiring the user to wait for the downloading to complete 

applications and, more particularly, to the streaming of before the presentation can begin. This delay can be con- 
real-time multimedia information over a communication siderable and is especially annoying when a user finds that 
network. he or she is only interested in a small portion of the overall 

2. Discussion of Related Art ^ presentation. 

Generally speaking, multimedia applications present ^ ™^ sophisticated model of producing information 
related media information, such as video, audio, music, etc., involves a server at one network site "streaming" the mul- 
on a presentation device, such as a computer having a timedia information over the network to a client at another 
display and sound system. Some multimedia applications site. The client begins to present the information as it arrives, 
are highly interactive, whereas other applications are far less ^^iher than waiting for the entire data set to arrive before 
interactive. For example, a game is a highly interactive beginning presentation. This benefit of reduced delay is at 
application in which the apptication must respond to many ^^e expense of increased complexity. Without the proper 
user inputs such as keyboard commands and joystick control, data overflow and underflow may occur, seriously 
movements, whereas viewing a vkleo clip is less interactive degrading the quality of the presentation, 
and may only involve start and stop commands. Moreover, Many modem multimedia applications involve the trans- 
multimedia applications may be directed to standalone fer of a large amotmt of information, placing a considerable 
single computer contexts, or they may be directed to load on the resources of the network, server, and client. The 
distributed, network-based contexts. use of network-based muhimedia apptications appears to be 

At a very high level of abstraction, following the 4^ growing. As computers become more powerful and more 

producer-consumer paradigm, any multimedia application people access network-based multimedia applications, there 

involves producing and consuming the related multimedia will be an increased demand for longer, more complicated, 

information. The above examples of highly interactive and more flexible multimedia applications, thereby placing even 

less interactive and standalone and network-based applica- larger loads and demands on the network, server, and client, 

tions differ in the manner in which the information is ^5 ^h® demand placed on servers by these ever-growing mul- 

produced and consumed and the complexity in controlling timedia apptications is particularly high, as individual serv- 

the production and consumption . ^^e called upon to support larger numbers of simuJta- 

For example, in PCs and other sUndalone contexts, the ^^^^ ^^^J^ is not uncommon even today for an Internet 

information need only be read off a local CD Rom or tiie »° ^^"^J^ thousands of smiultaneous chamiels. 

like, and thus, producing the information to the consumer 50 ^Wqu^^ntly there is a need m the art for a device, system, 

involves relatively predictable characteristics and requires '^^^^^^^ other thmgs, 

relatively simple control logic. In network-based contexts, cao t>andle longer, more complicated presentations; 

on the other hand, the information must be produced over utilize a network's resources more efficiently; and 

the network and is thus subject to the unpredictable char- utilize a server's and client's resources more efficiently, 

acteristics and intricacies of the network: data may be lost, 55 

performance may vary over time, and so on. ConsequenUy, SUMMARY 

the control logic may need to be relatively complicated. Iq short, the invention involves a new file format for 

In either the standalone or network-based context, con- organizing related multimedia information and a system and 

suming the information involves presenting the related device for, and method of, using the new file format. The 

information to the corresponding presentation components 60 invention eases the management and control of multimedia 

in a controlled manner and in real time. For example, to presentations, having various media streams, each of a 

provide intelligible audio-video ctips, the video data must be specific type, each specific type further classified by encod- 

provided to a video driver and audio data must be provided ing type, subtype, and encoding rate. Thus, with the 

to a sound card driver within ^ecified timing tolerances to invention, an application may support several instances of a 

maintain intra- and inter-stream synchronism. Intra-stream 6S particular media type, with each instance having different 

synchronism means that a given stream, such as audio, is characteristics. For example, the application may support 

presented in synchronism within specified time multiple audio streams, with each stream in a different 
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language, from which a user may select. Analogously, with 
the invention, the application may choose a given instance 
of a media stream based on the network's characteristics; for 
example, the application may choose an audio subtype that 
is encoded for a transmission rate that matches the network*s 
characteristics. The invention reduces a server's memory 
and processing requirements, thus allowing a server to 
simultaneously service more requests and support more 
channels. And, the invention dynamically adapts the medians 
streaming rate to use the network's resources more eflB- 
ciently while minimizing the effects of the adaptation on the 
quality of the presentation. 

The invention includes a system and device for, and 
method of, presenting multimedia information. In a client- 
server context, the invention includes a client that receives 
units of the multimedia information and presents the infor- 
mation on a presentation device. Each unit of information 
has an importance value assigned to it, which in an exem- 
plary embodiment is indicative of the unit's importance in 
relation to the quality of the presentation. The invention 
includes a mechanism for characterizing the performance 
capabilities of the system. For example, several conven- 
tional statistics may be gathered and analyzed concurrently 
with the streaming operation and before it begins.! The 
invention includes a mechanism for inferring network con- 
ditions from the characterized performance. The server may 
then stream the units of multimedia information to the client 
at a streaming rate and adapt the streaming rate of the 
streaming in response to the importance information and in 
response to the inferred network conditions. 

According to one aspect of the invention, the importance 
information that is pre-assigned is stored with each imit of 
information in the file. This importance information is 
indicative of the unit's importance to the perceived quality 
of the presentation 

According to another aspect of the invention, the adap- 
tation of the streaming rate includes throttling up the stream- 
ing rate if the inferred network condition indicates that there 
is sufEcient bandwidth to receive multiple copies of a given 
unit of information. 

According to yet another aspect of the invention, the 
adaptation of the streaming rate includes throttling down the 
streaming rate if the inferred network condition indicates 
that there is insufficient bandwidth to receive a given imit of 
information. Similarly, throttling down may occur if the 
inferred network condition indicates that there is network 
congestion. 

One of the performance charaaeristics gathered, believed 
to be novel, is the characterizing of a bit rate throughput 
value. This may be done, for example, by determining the 
data rate of a communications device, such as a modem, 
utilized by the client computer to communicate with the 
server computer. 

BRIEF DESCRIPTION OF THE DRAWINGS 
In the Drawing, 

FIG. 1 is a block diagram showing the novel multimedia 
file at a high level of abstraction; 

FIG. 2A is a block diagram showing the file header of the 
muhimedia file;. 

FIG. 2B is a block diagram showing the format of a file 
header preamble; 

FIG. 20 is a block diagram showing the format of a media 
instance descriptor; 

FIG. 3 is a block diagram showing the format of a media 
block; 
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FIG. 4A is a block diagram showing a generic media 
block directory format; 

FIG. 4B is a block diagram showing the format of a 
generic directory preamble; 

FIG. 4C is a block diagram showing the format of a 
generic packet descriptor; 

FIG. 4D is a block diagram showing the format of a 
generic media block body; 
IQ FIG. 4E is a block diagram showing the format of a 
generic packet; 

FIG. 5 is a block diagram showing the formal of an H.263 
media block; 

FIG. 6 is a block diagram showing the format of a MIDI 
^5 media block preamble; 

nG. 7 is a block diagram showing the format of a MIDI 
packet; 

FIG. 8 is a block diagram showing a system for creating 
a multimedia file; 

FIG. 9 is a block diagram showing a system embodying 
the invention in a cKeot-server context; 

FIG, 10 is a block diagram showing the client and server 
components of the system; 
25 FIG. 11 is a flow diagram showing an initial interaction 
between the client and the server; 

no. 12 is a flow diagram showing the streaming logic of 
the server; and 

FIG. 13 is a flow diagram showing the retransmit logic of 
the server. 

DETAILED DESCRIPTION 

In short, the invention involves a new file format for 

35 organizing related multimedia information and a system and 
device for, and method o^ using files organized according to 
the new format. The invention eases the management and 
control of multimedia presentations, having various media 
streams, each of a ^ecific type, each specific type further 

4Q classified by encoding type, subtype, and encoding rate. 
Thus, with the invention, an application may support several 
related audio streams, such as English and French, from 
which a user may select. Analogously, with the invention, 
the application may choose a particular instance of a media 

45 stream based on the network's characteristics; for example, 
the application may choose an audio stream that is encoded 
for a transmission rate that matches the network's charac- 
teristics. The invention reduces a server's memory and 
processing requirements, thus allowing a server to simulta- 

50 neously service more requests and support more channels. 
And, the invention dynamically adapts the media's stream- 
ing rate to use the network's resources more efSciently while 
minimizing the effects of the adaptation on the quality of the 
presentation. 

I. The File Format 

The new file format, among other things, allows various 
types and subtypes of multimedia information to be 
organized, maintained, and used as a single file. The file 
60 format simplifies the server by not requiring the server to 
know, for example, which video files are related to which 
audio files for a given apphcation, or to know how to locate 
and use related files, each with their own internal 
organization, corresponding method of access and process- 
es ing. 

The file format allows multiple instances of a single 
media type to be stored in the file. Multiple instances of a 



12/02/2003, EAST Version: 1.4.1 



5,928,330 

5 6 

single media lype may be desirable for supporting alternate subtype, for example, English audio, French audio, QCIF 

encodings of the same media type, for example, an audio video, CIF video, etc. Field 225 indicates an encoding rate 

segment in multiple languages. This flexibility allows a of the corresponding media block; for example, for video 

single file to contain, in effect, multiple versions of the same information, the encoding rate indicates the target data rate 

presentation. Each instance corresponds to a "presentation's 5 which the video information was encoded by a video 

worth" of information for that media type. For example, encoder, such as the video encoder described in the related 

with the audio media type, an instance may involve the ^-S- Patent appUcation Improved Video Encoding System 

entire soundtrack in French or the entire soundtrack encoded and Method", identified and incorporated above whereas for 

at a particular rate information, the encoding rate might mdicate one of 

a number of audio sampling rates. 

TTie file format allows media mstances to be added and 10 ^ explained below, the number contained in field 

deleted to a file. This feature allows the file to be updated as 215 is not necessarUy the same as the number of media 

new media types and new media segments are developed, 31^3^5 that will eventuaUy be involved in a presentation, 

without requuing modification of the server's or client's xhe file contains a number of potentially related media 

logic to support the newly-added instances. This flexibility streams, or instances, organized according to media type, 

makes it easier to modify, create, and maintain large, com- encoding type, subtype, and rate. A presentation, on the 

plicated multimedia presentations. other hand, will likely involve only a subset of the available 

The file format aUows the server to implement more media streams, typicaUy one instance of each of the plurality 

flexible and powerful presentations. For example, the server of media types. For example, a given presentation v«U likely 

could support multiple languages as various subtypes of an involve only one of the multiple audio (language) subtyped 

audio stream. In addition, the server could support multiple. instances that may be provided by a file organized according 

expected transfer rates. For example, a video media type ^J^^ f^""^^' '^f f ^ data encoded at 

may be implemented as a subtyped instance having pre- ^^^f.^» of cou^, a user may not be mterested m 

packetized video data encoded for a target transfer rate of f oomphment of media streams, e.g. the user may not 

28.8 kb/s or encoded for a target transfer rate of 14.4 kb/s. ^? interested m receiving audio, even if it is supported by a 

^ o rvwo ^JL w LKJi a toig^^i iiau^i».i im^ kji. x-t.-t xu//o. Dcpendmg upon the services supported by the server 

Moreover, when properly used by a server or other (more below), the actual media types and particular 

application, files organized according to the new format will instances of those media types involved in a presentation 

reduce the amount of memory and processor resources may be controUed by an end user, e.g., which language, and 

required to stream the file's contents to a client or the like. may also be controlled by the system, e.g., which encoded 

These advantages are further discussed below. rate of audio. 

The new file format 100 is shown at a high level of Once the media types and particular instances of those 

abstraction in FIG. 1. The file format 100 includes a file media types have been determined, for example, by being 

header 110 and a file body 120. In short, the file header 110 selected by the user or the server, the server will construct 

describes the file itself and the contents of the file body 120 data structures using information from the file header HO, 

and includes information used to locate data in the file body. 35 desoibed above, so that the server can index into and iterate 

The file body 120 includes more information used to locate over the data packets contained in the file body 120. 

data in the file body as well as inchiding the actual data used (Indexing and iteration logic are known) The server can also 

during a presentation. use the header's information to perform revision control and 

More specifically, the file header HO includes a file other known maintenance operations, discussed below when 

header preamble 210 and a number of media instance 40 describing an exemplary server. 

descriptors 220, shown in FIG. 2A. The file header preamble The data contained in the file body 120 is organized as 

210, shown in more detail in FIG. 2B, includes a field 211 contiguous media blocks 310, one media block for each 

containing a file signature, a field 212 containing the size of instance of a media type, as shown in FIG. 3. Each media 

the header, a field 213 containing the major version number, block 310 inchides a media blodt directory 320 and a media 

a field 214 containing the minor version number, and a field 45 blodc body 330. The media block directory 320 includes 

215 containing the number of media instances in the file. information that may be used to locate information in the 

(The major and minor version numbers are used by the client media block body 330, and the media block body 330 

and the server to determine if they are compatible, for includes the actual data that will eventually be presented, 

example, to determine whether they are communicating with This data is stored in the media block body 330 in pre- 

anexpectationof the same format of the file) The file header 50 padcetized form. "Pre-packetized** means that the data 

preamble 210 also includes reserved fields 216 and 217 to stored in media block body 330 is organized as discrete 

allow for future expansion of the preamble. packets of information that can be tran^orted without 

As shown in more detail in FIG. 2C, each media instance requiring any processing by the server to build the packets, 

descriptor 220 includes a variety of fields 221-225 which An exemplary embodiment, discussed below, pre-packetizes 

are used to describe and identify a media instance. Some of 55 the data so that it can be applied directly to the User 

the fields are used to describe various characteristics or Datagram Protocol (UDP) layer, which is part of the TCP/IP 

attributes about the instance's data that will be presented, protocol suite. (UDP and TCP/IP are known), 

whereas other fields are used to locate and select the data. The pre-packetization process is media instance specific. 

Field 221 indicating the offset, e.g., number of bytes, from For example, the G.723 audio encoding standard encodes an 

the beginning of the body to the corresponding media block. 60 audio stream into a stream of blocks, in which each block 

The media instance descriptor 220 also includes a field 222 represents 30 milliseconds of audio information. One 

indicating the media type of the corresponding media block, method of pre-packetizing the audio would be to form an 

for example, video, audio, MIDI, and other existing and audio packet for each G.723 block. A potentially more 

future media types. Field 223 indicates the encoding type of efficient method, however, merges many g.723 blocks into a 

the corresponding media block, for example, H.263, H.261, 65 packet, for example, 32 G.723 blocl^ to form an audio 

MPEG, G.723, MIDI, and other standard or proprietary packet representing 960 miUiseconds' worth of audio infor- 

encoding types. Field 224 indicates a corresponding mation. 
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Pre-packetizing video information, on the other hand, 
may benefit from dividing, rather than merging, presentation 
units. For example, under the H.263 video encoding 
standard, video information is encoded as a sequence of 
video frames (i.e., a frame being a presentation unit). 
Although a video packet may be formed to correspond to a 
single video frame, or presentation tmit, advantages may be 
attained by dividing the presentation unit into several pack- 
ets. In this fashion, a large video frame may be divided into 
several packets so that each video packet is limited to a 
predetermined, maximum packet size. 

By pre-packetizing the data, an appropriately designed 
server's processor load may be reduced by alleviating the 
processor from having to perform certain tasks such as 
constructing packets on-the-fly from the media information. 
With the invention, the server can simply read a packet from 
the file and pass it to a UDP layer of the protocol stack via 
a standard interface. 

In addition, by pre-packetizing the data, an appropriately 
designed server's memory requirements are by alleviating 
the server from having to keep recently transmitted packets 
available in a "packet window" in memory. Packet windows 
are conventionally used to hold recently-transmitted net- 
work packets in case they need to be retransmitted because 
they were lost in the network. The protocol being used 
dictates the required size of a packet window, but it is not 
UDCommoQ in modern systems to have windows that require 
on the order of 100 kb of memory (RAM). Given that each 
network channel requires a corresponding packet window 
and that it is not uncommon for current high-demand 
Internet servers to support upwards of 5,000 simultaneous 
channels (with foreseeable demand growing to over 20,000 
simultaneous channels in the near future), 512 Megabytes of 
expensive high-speed memory are needed just to support 
packet windowing. This requirement precludes many mod- 
era persona] computers, and other small systems, from 
operating as a server. In contrast, the invention obviates the 
need for the packet windows and thus allows smaller, 
lower-cost systems to potentially operate as servers. 

The organization of a generic media block 310 is shown 
in FIGS. 4A-E and defines the basic template for a media 
block. In short, the generic media block format describes 
certain features common to all media types and instances. As 
will be described below, specific media types, such as video, 
audio, and MIDI may need to "supplement" the generic 
template. 

A generic media block directory 400, shown in FIG. 4A, 
includes a directory preamble 410 and a number of packet 
descriptors 420. The directory preamble 410, shown in more 
detail in FIG. 4B, includes a packet count field 411, indi- 
cating the number of packets contained in the media block 
body and a presentation length field 413 indicating the total 
time duration of the presentation. A number of reserved 
fields 412 and 414 provide additional storage for media 
instance specific information (more below). 

A packet descriptor 420, shown in FIG. 4C, describes a 
single packet in the media block body. There is a one-to-one 
correspondence between packet descriptors 420 in the media 
block directory and packets in the media block body. Each 
packet descriptor 420 includes a packet offeet field 421 
indicating the ofi&et, e.g. bytes, from the start of the media 
block to the corresponding padcet, a time stamp field 422 
indicating the start time for the packet relative to the start 
time of the presentation, an importance field 423 indicating 
the relative importance of the corresponding packet (more 
below), and a packet length field 425 indicating the length, 
e.g. bytes, of the corresponding packet. 
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A generic media block body 430, shown in FIG. 4D, 
includes a number of packets 440. This number represents 
the number of padcets in the stream, or instance. 

A packet 440, shown in more detail in FIG. 4E, includes 
5 a channel niunber field 441, which is reserved for a currently 
undefined future use, and an importance field 442 indicating 
the relative importance of the packet to the perceived quality 
of the presentation (more below). Each packet 440 also 
includes fields 443-444 that may be used for segmentation 
10 and reassembly (SAR) of presentation units. As outlined 
above, depending on the media instance and other 
circumstances, advantages may be attained by dividing 
video frames into multiple packets. To accomplish this, field 

443 may be used to indicate the sequence number of the 
15 given packet relative to the multiple packets composing the 

presentation unit, e.g., video frame. A total segments field 

444 indicates the total number of segments. 

The packet 440 frirther includes a packet length field 445 
indicating the number of bytes of data contained in the 
packet, a sequence number field 447 which is reserved for a 
currently undefined future use, a time stamp field 448 
indicating the start time for the packet relative to the start of 
the presentation, and a data field 460 containing the media 
data that will be eventually presented. This media data is 
encoded in a predetermined format, according to the media 
type, encoding type, subtype, and rate (see FIG. 2C); the 
encoding format may be standardized, in the process of 
being standardized, or proprietary. A reserved field 449 
provides additional storage for media type specific informa- 
tion. 

Each packet, like its descriptor, includes pre -assigned 
importance information, indicative of the relative impor- 
tance of the packet with respect to the quality of the eventual 
presentation. As will be explained below, some media 
frames are highly important; their absence from the presen- 
tation may make it unintelligible. Other frames are less 
important; their absence may be barely noticed or may be 
"concealable.** ("concealing" is known) 

^ As will be described below when discussing an exemplary 
server, importance information may be used by both the 
server and the client. For example, the server may use the 
information to intelligently adapt its streaming characteris- 
tics to better utilize the network's resources. The server may 

45 "drop" packets from being sent, when needed, or send 
multiple copies of packets, if beneficiaL On the other hand, 
the cUent may use the information to inteUigendy maintain 
synchronism. The client may drop relatively unimportant 
video packets to maintain synchronism with the audio 

50 stream being presented. 

Regarding the format of specific media blodts, video 
information may be formed into a media block, having data 
in accordance with the H.263 format. H.263 specifies that 
video streams will include "I" frames, "F' frames, and "PB" 

55 frames. The "I" frames are independent frames in that they 
represent a complete image, with no dependencies on the 
prior history of the video image. Therefore, I frames can be 
considered as specifying the "state" of a picture. The "P" and 
"PB" frames^ on the other hand, specify the changes to a 

60 current picture, as defined by an I frame and intervening P 
and PB frames. Therefore, P and PB frames are akin to "state 
changes," rather than states. A typical video stream would 
involve an I stream followed by P and PB frames, followed 
by more I, P, and PB frames. The processing of I, P, and PB 

55 frames is known. 

The format of the H.263 media block is generally the 
same as the generic media block, described above with 
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reference to FIGS. 3-4E. However, the H.263 media block contiguous portion of a body 120 to contain the media block, 

does supplement the generic format in certain areas. The The header 110, including the preamble 210 and media 

H.263 directory preamble 500, shown in FIG. 5, for instance descriptors 220 (see FIG. 2) are programmed with 

example, differs from the generic template by including a the values, described above, as part of the final assembly of 
field 512 indicating the maximum frame size in the presen- 5 the file 850, Much of the information, such as the importance 

tation and by including a field 514 indicating the number of information, is supphed by the encoders 825 and 830. 

I-frames in the H.263 video stream, or instance. jhe multimedia file 850 is easily modified to add or delete 

Audio information may be formed into a media block, media blocks, for example, to add additional languages, data 

having data in accordance with the G.723 format. The rates, or backgroimd music. To add a media block, the file 
format of the H.263 media block is identical to the generic 10 header is edited to include a new media instance descriptor, 

media block, and contains no additional information. and the media block is added into the file body. To delete a 

MIDI information may be formed into a media block, media block, the file header is edited to delete the corre- 
having data in accordance with a MIDI format. The format spending media instance descriptor, and the media block is 
of a MIDI media block is generally the same as the generic removed from the file body, 
media block. However, the MIDI media block does supple- • r • 
ment the generic format in certain areas. The MIDI directory Overview of Device and System for, and 
preamble 600, shown in FIG. 6, for example, differs from the Method of, Presentmg Multimedia Information 
generic template by including a field 612 indicating one of in addition to the above, the invention provides a device 
a plurality of MIDI formats, a field 614 indicating the and system for, and method of, presenting the multimedia 
number of tracks contained in the MIDI stream, and a information in a manner that efficiently uses network band- 
divisor field 616 used to specify the tempo of the MIDI width and that efficiently uses a server's processor and 
stream. The MIDI packet 700, shown in FIG. 7, differs from memory resources. As will be explained below, the device, 
the generic packet 440 (see FIG. 4E) by including a field 749 system, and method allow a server to support more simul- 
indicating the start time of the MIDI packet relative to the taneous channels and allow a server to intelligendy and 
start of the presentation and by including a field 750 indi- dynamically adapt its streaming characteristics. More 
eating the relative end time of the MIDI packet. specifically, by using the pre-packetized information, the 

^ T--. A J- . XT . server needs lessprocessor and memory resources to support 

II. Creating Files Accordmg to the Novel Format . , . ^ . . . ^ ... 

* ^ a channel vis-a-vis conventional arrangements, and thus, a 

It is expected that developers will create multimedia files server having a given amount of processor and memory 

in the format described above using various methods and resources can support more channels by using the invention, 

systems. FIG. 8 shows but one simple example. For In addition, by using the importance information pre- 

simplidty, the exemplary system 800 creates a file 850 with assigned to each packet, a server can dynamically adapt its 

an audio stream(s) and video stream(s) only, but skilled streaming characteristic to send multiple copies of a packet 

artisans will appreciate the relevance of the teachings to if the situation warrants, or to eliminate certain packets from 

other media types, e.g., MIDI, etc. being sent if the situation warrants. 

Media components, such as VCR 805, camera 810, and A high-level architectural diagram of a system, embody- 

microphone 815, generate media information, which is ing the invention in a client-server context, is shown in FIG. 

received by a sound and video capture system 820, for 9. In this context, a client 910 communicates with a server 
example, available from AVID Technologies or Adobe, Inc. ^ 920 via a communication network 930. The client may, for 

The capture system 820 may store the captured audio and example, include a PC, and the server may include a 

video information in a proprietary, but known, format. UNIX-based workstation, minicomputer, or tiie like. The 

The captured information is applied to video encoder 825 communication network 930 may utilize a variety of known 

and audio encoder 830, which encode the captured iofor- physical mediums and protocols, such as modem links, 
mation into frames according to predetermined formats, 45 LANs, using the TCP/IP suite or the like. The communica- 

such as H.263 for video and G.723 for audio. For example, tions network 930 allows control information 931 and data 

the video encoder 825 may use software-based logic to do 932 to be exchanged between the client 910 and the server 

the encoding from the format of the capture system 820 into 920. 
the predetermined format, e.g., H.263. An exemplary 

encoder is described in the U.S. patent application entided 50 ^ ^^^^ ^P^^^ Example of the Device, 

"Improved Video Encoding System and Method", identified System, and Method 

and incorporated above. Analogous software-based process- One specific example of a ctient-server context in which 

ing could be applied to the audio information, originating the invention may be practiced is the Internet. In the Internet 

from system 820. context, as will be appreciated by users familiar with the 
The encoded video, originating from encoder 825, and the 55 Intemct, the client 910 might include a known browser 

encoded audio, originating from encoder 830, are apptied to application for navigating access to various severs 920 

respective packetizers 835 and 840. Each packetizer connected to the Internet. Each server might use known web 

arranges the encoded data in pre-packetized form with the server applications to display web pages, which might 

necessary packet information, e.g., importance, packet include text and graphical icons, among other things. Some 
length, and time stamp (see FIG. 4E). For the most part, each 60 of the icons might be "active" in which case cHcking the 

video frame will have its own corresponding packet, such as icon would cause a video cfip to be presented to the client, 

a UDP packet, but as described above with regard to FIG. The mechanisms for causing the presentation of the video 

4E, a frame may be divided up into several packets, for clip are discussed below. 

example, if the video packet is very large. Analogous packet An exemplary arrangement of using the invention in the 
processing is apptied to the audio frames. 55 Internet context is shown in FIG. 10. The client 910 includes 

The output of the packetizer is written to file 850, ioi conventional browser appUcation 1010, and the server 920 

example, on a CD ROM, such that each media type uses a includes conventional web server application 1030 and web 
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pages 1031, all of which will not be discussed in detail The interaction ends in step 1199, and the server is ready to 

because they are known. The web browser application 1010 begin streaming the media data from the file, 

cooperates with novel multimedia client application 1020 to b. Streaming 

initiate the processing of file 850, described above. In turn, In an exemplary embodiment, the streaming process is 

muhimedia client application 1020 cooperates with novel 5 performed using the UDP transport protocol of the TCP/IP 

multimedia server application 1040 to produce the mulli- protocol suite. The UDP protocol is an 'Hmreliablc protocor 

media file 850 from the server. The interaction and coop- in that it assigns responsibility of rehable transmission to 

eration of the above entities are ftirther described below. higher layers. Thus, the client logic and the server logic are 

a. Set Up responsible for detecting lost packets and the like and 

The web browser application and the web server appli- lO performing the appropriate action. This situation is unlike 

cation interact through a known standard interface, which other transport protocols such as TCP in which detection of 

will be described briefly. The web browser application is lost packets and responding to such detection is handled by 

used to view web pages on various servers in the network. the transport and lower layers. 

A web page may allow access to certain files stored on the The streaming logic of the server is shown at a high level 

server, including multimedia files such as file 850 (sec FIG. is of abstraction in FIG. 12. The logic starts at step 1200 and 

8). To view a file, the web browser requests a Universal proceeds to step 1210 where it is determined whether 

Resource Locator (URL) from the web server application, streaming should continue. This step includes determining 

and the web server application responds with a message whether all of the packets for the requested presentation 

which includes a MIME type which specifies the location have already been streamed, whether an error has occurred, 

and type of the file. Based on the file type, the web browser 20 whether insufficient bandwidth exists to make continued 

application may need to invoke a "helper" application, presentation impractical, whether the user has requested the 

which is used i^ecifically to handle files having the specified presentation to stop, whether the user has disconnected from 

file type. In the case of muhimedia file 850, the web browser the server, and the like. If streaming is to stop, the logic 

application invokes the novel multimedia client application, proceeds to step 1299, which ends the flow. If streaming is 

which initiates an interaction with the novel multimedia 25 to continue, the logic streams a time-slice's worth of packets 

server appUcation to produce the multimedia file. in step 1220. The actual logic of this step is described below. 

An exemplary initial interaction between the multimedia After streaming a time-slice's worth of the packets in step 

client application 1020 and the multimedia server applica- 1220, the logic proceeds to step 1230 where the logic 

tion 1040 is described below with reference to FIG. 11. The requests to go to sleep, or go idle, for a time quanta 

interaction begins in step 1100 and proceeds to step 1110, 30 corresponding to the streaming rate. For example, if the 

where the multimedia client application 1020 sends a media logic is streaming audio to correspond to 960 ms slices of 

request message to the multimedia server application 1040 presentation, step 1220 will stream 960 ms worth of 

specifying the desired media types to be produced, and information, and step 1230 wiU request to go to sleep until 

specifying a version number of the multimedia client appli- it is time to stream another 960 ms worth of information, 

cation and a port number to which the multimedia server 35 Break 1235 indicates that a break in the flow of control from 

application should direct commtmication. In step 1120, the step 1230 back to step 1210 may occur (more below), 

multimedia client application 1020 sends a message to the Assuming no break in the flow, the above logic will continue 

multimedia server application 1040 specifying a desired rate to loop and stream media packets, uintil one of the 

of transmissioiL The desired rate of tran^ission may be conditions, discussed above with regard to step 1210, is met. 

determined by the client, for example, by determining the 40 As alluded to above, modem networks lose packets for 

communication rate of an attached communication device variousreasons.If the client detects a lost packet it may send 

such as a modem (more below). The interaction then pro- to the server a retransmission request for the missing packet, 

ceeds to step 1130, where the multimedia client appHcatioo Given that the exemplary embodiment uses UDP, this 

1020 sends a "go" message to the server ^plication 1040, retransmission request is handled by the server logic 1040, 

informing the multimedia server application that the initial 45 rather than the lower layers of the protocol, 

client messages have been sent, and therefore allowing the In response to a retransmission request, the logic 1040 

server to determine whether or not all messages were will break at point 1235 and proceed to step 1240 which will 

received. retransmit the packet. This logic, like the logic of step 1220, 

Upon receipt of the "go" message, the interaction pro- is discussed below. The logic then proceeds to step 1250 
ceeds to step 1140, where the multimedia server application 50 which determines whether there is more time during which 
1040 sends a media response message to the multimedia the logic should remain asleep. If so, the logic proceeds to 
client application 1020 specifying, among other things, the step 1260, where the logic goes back to sleep for the 
time of day as determined by a reference clock in the server remaining quanta set in step 1230. If not, the logic proceeds 
or network. In step 1150, the multimedia server application directly to step 1210 to continue streaming. 
1040 sends a message to the multimedia client application, 55 The streaming and retransmit logic of steps 1220 and 
which immediate responds in step 1160 with another mes- 1240 are described below. A few general words arc war- 
sage. The server application 1040 uses the above "echo" to ranted before the detailed discussion. First, the streaming 
calculate a round-trip delay of the network. In step 1170, the and retransmit steps may determine not to transmit certain 
multimedia server application 1040 sends a configuration packets at all. Second, the streaming and retransmit logic 
message to the multimedia client application 1020 60 dynamically adapt the streaming characteristics of the server 
specifying, among other things, the sizes and relationships logic by either eliminating certain packets firom being 
of the information to be produced. The multimedia client transmitted, or by sending multiple copies of other packets 
application 1020 uses the configuration information to con- to increase the probability that the client wiU receive a 
trol its consumption of the information produced, or complete sequence of packets. This dynamic adaptation is 
streamed, by the server. Finally, in step 1180, the multimedia 65 based on information contained in the packets and their 
server application 1040 sends a "go" message to the client descriptors and on network statistics gathered before and 
application 1020, indicating the end of the initial interaction. during the streaming operation. More specifically, the sta- 
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tistics are used to create "inferences" of the network state, 
which are used by the logic, described below, to adapt its 
behavior. 

The server logic, as a general matter, indexes into and 
iterates over the various streams involved in the presenta- 
tion. (Indexing and iteration are known) When retransmis- 
sion is involved, the server logic indexes into the file to 
select the appropriate packets, while maintaining the itera- 
tion pointers for the normal streaming loop, i.e., steps 
1210-1235 (see FIG. 12). 

The presentation is divided into time quanta, which is 
typically dictated by the audio presentation rate and quanta, 
e.g., 960 ms. The server logic selects the appropriate 
packets, tising indexing and iteration, &om the file contain- 
ing the various streams of the presentation. An exemplary 
embodiment selects the packets to be transmitted by com- 
paring a clock value maintained by the server logic and in 
rough synchronism with the client with the time stamps 422 
(see FIG. 4c) corresponding to the packetized data. Alter- 
native arrangements include selecting the related packets 
based on a ratio relating the various streams; for example, 
several audio packets may correspond to a single video 
packet. Similarly, if other file organizations are used, for 
example, prc-interleaved audio and video, the server logic 
would select the appropriate segments. 

Once the next packets to potentially send are identified, 
the logic determines whether they should be sent and, if so, 
how many copies should be sent. As outlined above, this is 
determined on statistic-based inferences (more below) and 
upon pre-assigned importance fields of the packets. An 
exemplary embodiment uses the number 1 for highest 
importance packets, with higher numbers corresponding to 
lower importance. As discussed above, these importance 
fields are assigned during the creation of the file, for 
example, with the encoder outlined above. Some encodings 
that arc expected to be followed by all implementations, and 
which are followed by the exemplary embodiment, are that 
audio padcets and I firames of video will be assigned an 
importance of 1, i.e., highest priority. In this fashion, as will 
be evident from the description below of the detailed logic 
of FIG. 13, these packets will always be transmitted and 
retransmitted when requested. 

The logic of FIG. 13, described below, utilizes statistics 
gathered on the network. Ad exemplary embodiment imple- 
ments the statistics gathering function in the client 910, 
because it wiU be the only entity capable of gathering certain 
statistics. If a different set of statistics is utilized in other 
embodiments, it may be beneficial to place the re^onsibility 
of gathering statistics with the server. Moreover, either the 
client 910 or the server 930 could perform the analysis on 
the statistic to create inferences or the like. In the former 
case, the client would gather and analyze the statistics and 
send the inferences to the server as control packets for use 
by the logic of FIG. 13. In the latter case, the client would 
send the statistics to the server, which would then analyze 
the statistics to create the inferences used by the logic of 
FIG. 13. 

The currently envisioned set of statistics necessary to 
create the inferences used by the logic of FIG. 13 include the 
following: 

1. bit rate throughput; 

2. network jitter; 

3. round-trip delay; and 

4. percentage and distribution of packet loss. 

Statistics 2-4 may be obtained with conventional tech- 
niques. Consequently, the gathering of these statistics are not 
discussed. 
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Statistic 1, however, needs to implement novel function- 
ahly as not all communication devices, e.g., certain model 
modems, are capable of providing these statistics during the 
streaming operation. An exemplary client determines 
throughput by monitoring and analyzing UARTintermpts to 
determine, or infer, the modem's throughput measured as a 
bit rate. 

As outlined above, the transmission and retransmission 
steps of FIG. 12 are highly abstracted. The detailed logic of 
these steps is described with reference to FIG. 13. 

The logic begins at step 1300 and proceeds to step 1310, 
where an inference is checked to see if there is a modem or 
similarly related network performance problem. This infer- 
ence is created fi:om a detected decrease in throughput (e.g., 
bit rate) without a corresponding detection of significant 
packet loss. If there is such a problem, the logic proceeds to 
step 1320 where a "throttle down" operation is performed, 
which sets a Sag indicating that a packet with importance 
fields equal to 5 should not be sent, or more specifically that 
a packet whose descriptor has an importance field of 5 
should not be sent. As stated above, the assignment of fields 
depends on the encoding of the file, but some examples of 
expected packets having assignments equal to 5 include the 
last P or PB firames immediately before an I frame of video 
in a video stream. These firames are relatively less important 
because the following I fi'ame will reset the state and there 
are no intervening states depending on these P frames. 

If there is no modem or analogous problem in step 1310, 
the logic proceeds to step 1330 to determine if there is a 
network-based problem. This inference is created from 
detecting a significant packet loss. If there is no network- 
based problem, the logic proceeds to step 1340, which 
indicates that no throttling should occur, in short, that the 
network is performing adequately and that the streaming 
characteristics should tie normal and not modified. 

If there is a network problem in step 1330, the logic 
proceeds to step 1350 to determine whether or not the 
network problem involved random packet loss. If not, mean- 
ing that the packet loss involved a significant number of 
contiguous or nearly-contiguous packets in a small time 
firame, the inference is that the network is experiencing 
congestion, and the logic proceeds to step 1360, where a 
throttle down operation is performed to set a flag that 
packets with importance 4 or higher should not be sent. 

If step 1350 indicates that the problem is random packet 
loss, the logic proceeds to step 1370, where a "throttle up" 
operation is performed if the statistics indicate that the 
network has available bandwidth. The available bandwidth 
is determined as part of the initial characterization of the 
network, discussed above with regard to FIG. IIB and 
which may be continually updated as part of the concurrent 
network characterization. More particularly, flags will be set 
to indicate that packets having importance less than 3 should 
be sent multiple times, e.g., in duplicate. 

Steps 1320, 1340, 1360, and 1370, then proceed to step 
1380 which sends the time quanta's worth of packets, 
discussed above, in consideration of the various flags set 
during throttle up and down steps. The logic then ends in 
step 1399. 

The throttle down operations eliminate certain packets 
firom being transmitted or retransmitted. In short, the under- 
lying philosophy is that, in these situations, sending the 
packet will be to no avail or will exacerbate the detected and 
inferred problem. For example, if there is low throughput 
because of a modem problem, it does not make sense to send 
all of the packets because the client will likely not receive 
them in suflScient time for synchronized presentation. 
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Moreover, if the network is congested or the like, resending 
packets will only increase the congestion and worsen per- 
formance. 

The throttle up operations transmit, or retransmit, certain 
packets if bandwidth is available. In short, the underlying 
philosophy is that, if the network resoiu-ces are available, 
then they should be utilized to increase the likelihood that 
the presentation will include more of the media information 
(i.e., less likely to drop packets for synchronization opera- 
tions at the client*s end). 

In both cases, the throttling is performed based on the 
importance information in the packet, which as outlined 
above indicates the importance of the packet to the quality 
of the presentation. Thus, less important packets are elimi- 
nated if the situation dictates, and more important packets 
are repeated if the situation dictates. The underlying effect is 
that the streaming characteristics are dynamically adapted in 
consideration of the importance of the information to the 
quality of the presentation. 

Skilled artisans will appreciate that the assignment of 
importance fields may change and that other inferences than 
those discussed above are easily incorporated into the above 
logic. Moreover, the logic may also be adapted to dynami- 
cally change a streaming subtype if the performance situa- 
tion warrants. 

Skilled artisans will also appreciate that the above archi- 
tecture may easily incorporate special effects processing, the 
logic of which will be evident, given the above description. 
For example, the server may provide fast forward and 
reverse functions. In such cases, the streaming logic, for 
example, could consider 1 frames only and ignore the P and 
PB frames. 

Skilled artisans will also appreciate that, although the file 
was particularly discussed in relation to UDP-ready packets, 
the invention is applicable to other protocols, other layers 
than transport, and other segmentations of information, e.g., 35 
fixed size cells. 

Skilled artisans will also appreciate that, although the 
media blocks were described as a contiguous arrangement, 
the invention is applicable to a single media block contain- 
ing multiple media types arranged in a pre-interleaved 40 
format. 

Skilled artisans will also appreciate that, although the file 
format was discussed in which media units, such as large I 
frames, are sub-divided, the invention is also applicable to 
merging smaller frames into larger packets. 45 

Skilled artisans will also appreciate that, although the 
characterization of the network was discussed as a general 
characterization, the invention is also applicable to charac- 
terizing the network on a channel-by-channel basis and by 
testing the network with a test audio stream and video 50 
stream, for example. 

The present invention may be embodied in other specific 
forms without departing from its spirit or essential charac- 
teristics, the described embodiments are to be considered in 
all respects only as illustrative and not restrictive. The scope 55 
of the invention is, therefore, indicated by the appended 
claims rather than by the foregoing description. All changes 
which come within the meaning and range of equivalency of 
the claims are to be embraced within their scope. 

What is claimed is: 60 

1. A system for presenting streaming multimedia infor- 
mation of a multimedia file comprising a phirafity of units of 
information, comprising: 

A) a client for receiving units of the multimedia infor- 
mation and presenting the information on a presenta- 65 
tion device as it is received, without waiting for the 
entire file to be received before beginning presentation; 



B) means for assigning importance information to indi- 
vidual units of the multimedia information; 

C) means for characterizing the performance capabilities 
of the system; 

D) means for inferring network conditions from said 
characterizing means; and 

E) a server for streaming the units of multimedia infor- 
mation to the client at a streaming rate, the server 
including adaptation means for adapting the streaming 
rate of the streaming in response to said importance 
information and in response to said inferred network 
conditions. 

2. The system of claim 1 wherein the client means is a 
cUent computer at a first network site and the server means 
is a server computer at a second network site and wherein 
the server means streams the information over a computer 
network connecting the client and server computers. 

3. The system of claim 2 wherein the cUeot and server 
computers communicate over the network in accordance 
with the UDP transport protocol. 

4. The system of claim 1 wherein the multimedia infor- 
mation is recorded as a file on a non-volatile storage. 

5. The system of claim 4 wherein the means for assigining 
the importance information comprises means for storing a 
pre-assigned importance value from a plurality of impor- 
tance values to each unit of multimedia information in the 
file. 

6. The system of claim 1 wherein the adaptation means 
includes means for throttling up the streaming rate if the 
infened network condition indicates that there is sufficient 
bandwidth to receive multiple copies of a given unit of 
information. 

7. The system of claim 1 wherein the adaptation means 
includes means for throttling down the streaming rate if the 
inferred network condition indicates that there is insufiScient 
bandwidth to receive a given unit of information. 

8. The system of claim 1 wherein the adaptation means 
includes means for throttling down the streaming rate if the 
infened network condition indicates that there is network 
congestion. 

9. The system of claim 1 wherein the importance infor- 
mation is indicative of the unit's importance to the perceived 
quality of the presentation. 

10. The system of claim 1 wherein the means for char- 
acterizing performs an initial characterization of the system 
before the server begins streaming the imits and wherein the 
adaption means is responsive to the intial cfaaracterizadon to 
select an initial streaming rate. 

11. The system of claim 1 wherein the means for char- 
acterizing performs characterization concurrently with the 
streaming performed by the server means and wherein the 
adaption means dynamically adapts the streaming rate in 
response thereto. 

12. The system of claim 2 wherein the means for char- 
acterizing comprises means for calculating a bit rate 
throughput value. 

13. The system of claim 12 wherein the means for 
calculating the bit rate throughput value comprises means 
for determining the data rate of a communications device 
utilized by the client computer to communicate with the 
server computer. 

14. The system of claim 13 wherein the means for 
determiniag the data rate of the communications device 
utilized by the client computer comprises means for mea- 
suring a periodic rale at which discrete units of data are sent 
from the communications device to the client computer. 

15. The system of claim 13 wherein the ctient computer 
comprises means for reporting the data rate to the server 
computer. 
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16. The system of claim 2 wherein the means for char- 
acterizing comprises means for calculating a percentage and 
distribution of packet loss. 

17. The system of claim 16 wherein the means for 
calculating the percentage and distribution of packet loss 
comprises means for detecting missing units of information 
from the units of information received by the client com- 
puter. 

18. The system of claim 17 wherein the client computer 
comprises means for reporting the percentage and distribu- 
tion of packet loss to the server computer. 

19. The system of claim 2 wherein the means for char- 
acterizing comprises means for calculating a round trip 
delay of the computer network. 

20. The system of claim 19 wherein the means for 
calculating the round trip delay of the computer network 
comprises: 

A) means for transmitting a first message from the server 
computer to the client computer, 

B) means for transmitting a second message from the 
client computer to the server computer upon receipt of 
the first message by the client computer; and 

Q means for measuring an amount of time between 
transmitting the first message and receiving the second 
message upon receipt of the second message by the 
server computer 

21. The system of claim 19 wherein the means for 
calculating the round trip delay of the computer network 
comprises: 

A) means for transmitting a first message from the client 
computer to the server computer; 

B) means for transmitting a second message from the 
server computer to the client computer upon receipt of 
the first message by the server computer; and 

C) means for measuring an amount of time between 
transmitting the first message and receiving the second 
message upon receipt of the second message by the 
client computer. 

22. The system of claim 2 wherein the means for char- 
acterizing comprises means for measuring an amount of 
jitter in the computer network. 

23. The system of claim 22 wherein the means for 
measuring the amount of jitter in the computer network 
comprises means for measuring a time span between the 
receipt of successive units of information by the client 
computer. 

24. The system of claim 23 wherein the client computer 
comprises means for reporting the amount of jitter to the 
server computer. 

25. The system of claim 2 wherein the computer network 
is capable of losing units of information appUcd to it. 

26. The system of claim 25 wherein the client computer 
comprises: 

A) means for detecting that a unit of information was tK>t 
received; and 

B) means for informing the server computer of the lost 
unit of information. 

27. The system of claim 26 wherein each unit of infor- 
mation comprises a pre-assigned sequence number and 
wherein the means for detecting that a unit of information 
was not received comprises means for detecting that the unit 
of information was received out of sequence. 

28. The system of claim 27 wherein the means for 
informing the server computer of the lost unit of information 
comprises means for reporting the sequence number of the 
lost imit of information to the server computer. 
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29. A server device for streaming to a client device a 
multimedia file in a computer network, the multimedia file 
comprising a plurality of units of information, each unit 
having importance information pre-assigned to the unit, the 
server device comprising: 

A) a computer-readable medium for providing the mul- 
timedia file; and 

B) a multimedia server application for reading the units of 
information from the computer-readable medium and 
for streaming the units of information onto the com- 
puter network, the streaming means including adaption 
means for adapting the streaming rate of the streaming 
in response to network conditions and in response to 
the importance information. 

30. The server device of claim 29 wherein the means for 
streaming the units of information onto the computer net- 
work comprises: 

A) means for determining whether all units of the plural- 
ity of units of information have been streamed; 

B) means for terminating streaming the method when all 
units of information have been streamed; 

C) means for streaming a number of imits of media 
information when less than all units of information 
have been streamed; and 

D) means for delaying for an amount of time calculated so 
as to maintain a bit rate of the stream at or below a 
predetermined bit rate. 

31. The server device of claim 30 wherein the computer 
network is capable of losing units of information applied to 
it. 

32. The server device of claim 31 wherein the multimedia 
server application further comprises means for deciding 
whether or not to retransmit a lost unit of information. 

33. The server device of claim 32 wherein the means for 
deciding whether or not to retransmit the lost unit of 
information comprises: 

A) means for obtaining a throughput rate of the computer 
network; and 

B) first means for decreasing the streaming rate when the 
throughput rate of the computer network has decreased. 

34. The server device of claim 32 wherein the means for 
deciding whether or not to retransmit the lost unit of 
information comprises: 

A) means for determining whether a network-based prob- 
lem exists; 

B) means, responsive to a status indicating a network- 
based problem, for checking whether the network- 
based problem is caused by random losses in the 
computer network; and 

C) first means, cooperating with the means for checking 
whether the network-based problem is caused by ran- 
dom losses, for increasing the streaming rate when the 
network-based problem is caused by random losses in 
the computer network. 

35. The server device of claim 32 wherein the means for 
deciding whether or not to retransmit the lost unit of 
information comprises: 

A) means for determining whether a network-based prob- 
lem exists; 

B) means, responsive to a status indicating a network- 
based problem, for checking whether the network- 
based problem is caused by random losses in the 
computer network; and 

C) second means, cooperating with the means for check- 
ing whether the network-based problem is caused by 
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random losses, for decreasing the streaming rate when 42. The method of claim 41 wherein the computer net- 

the network-based problem is caused by other than work is capable of losing units of information applied to it. 

random losses in the computer network. 43. The method of claim 41 wherein the method includes 

36. The server device of claim 32 wherein the means for the step of the client detecting that a unit of information was 
deciding whether or not to retransmit the lost unit of 5 not received and the client informing the server of the lost 
information comprises: unit of information. 

A) means for obtaining a throughput rate of the computer 44. The method of claim 43 wherein the step of the client 
network; detecting that a unit of information was not received 

B) first means for decreasing the streaming rate when the includes the step or the client monitoring sequence numbers 
throughput rate of the computer network has decreased; residing in each streamed unit of information and detecting 

C) means for chedcing the status of the computer network that a unit of information has a sequence number out of 
when the throughput rate of the computer network has sequence. 

remained constant or increased; 45. The method of claim 44 wherein the step of informing 

D) means, responsive to a status indicating a network- js the server of the lost unit of information includes the step of 
based problem, for checking whether the network- reporting the sequence number of the lost unit of informa- 
based problem is caused by random losses in the tion to the server. 

computer network; 46. The method of claim 45 wherein the method includes 

E) first means, cooperating with the means for checking the step of the server deciding whether or not to retransmit 
whether the network-based problem is caused by ran- 20 the lost unit of information. 

dom losses, for increasing the streaming rate when the 47. The method of claim 46 wherein the step of deciding 

network-based problem is caused by random losses in whether or not to retransmit the lost unit of information 

the computer network; and comprises the steps of: 

F) second means, cooperating with the means for check- A) checking a throughput rate of the computer network; 
ing whether the network-based problem is caused by 25 ^ throughput rate of the computer network has 
random losses, for decreasmg the streammg rate when decreased, decreasing the streaming rate; 

the network-based problem is caused by other than ^ * 

random losses in the computer network. Q ^ throughput rate of the computer network has not 

37. The server device of claim 36 wherein the first and decreased, checking the status of the computer net- 
second means for decreasing the streaming rate comprises work; 

means for dropping at least one of the plurality of units of D) if the status of the computer network indicates that a 

information. network-based problem exists 

38. The server device of claim 37 wherein the means for Dl) checking whether the network-based problem is 
dropping comprises means for determining an importance of caused by random losses in the computer network; 
each unit of information. jj2) if the network-based problem is caused by random 

39. The server device of claim 38 wherein the means for i^^^ in computer network, increasing the 
determining the importance coniprises means for referenc- streaming rate- and 

ing an importance value pre-assigned to each unit of infor- d3) if the netwo'rk-based problem is caused by other 

, . , t . . . . An than random losses in the computer network, 

40. The server device of claim 36 wherein the first means ^ ^ « „• ♦u . • 

.... . decreasing the streamme rate. 

for increasing the streammg rate comprises means for trans- ™_ °, r 1 - >*S i_ • . r. j 

, r * 1 * c L f 48. The method of claim 47 wherem steps B and D3 

mittmg multiple copies of at least one of the plurality of - , ^ . j ^ : ^ t r 

units of information. mcludes the step of droppmg one or more of the plurahty of 

41. A method of streaming a multimedia file, comprising of information ^ . ^ . ^ 

a plurality of units of information, each unit having pre- 45 49. TTie method of clann 48 wherein the step of droppmg 

assigned importance information, from a server to a cKent ^/^^ °^ &'^^^y information mcludes 

over a computer network so that the client may present the ^P, determinmg an importance of each umt of 

multimedia file as the units of information are received, the I^r^il'^"* .. 

.u J • • *u * c 50. The method of claim 49 wberem the step of deter- 

method comprising the steps of: . " " ..lauu ^^p- wucitiiu ui. utiti 

^ . , , , , . . - . - . - .50 mimng the importance of each imit of mformation mcludes 

A) Ihc server obtaining the units of information from the „f referencing an importance value pre-assigned to 

^' each unit of information. 

B) the server streaming the units of information onto the 51, The method of claim 47 wherein the step of increasing 
computer network to the client at a streaming rate; and the streaming rate includes the step of transmitting multiple 

C) the server adapting the streaming rate of the streaming 55 copies of one or more of the plurality of units of information, 
in response to network conditions and in response to 

the importance information. ***** 
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[57] ABSTRACT 

A wireless communication unit for a wireless communica- 
tion system transmits and receives video, audio and data 
signals within an RF bandwidth. The RF bandwidth is 
allocated among the video, audio and data signals to allow 
the video, audio and data signals to fit within the RF 
bandwidth. The allocation is performed by buffering the 
signals, making priority assignments to each of the buffered 
signals, and transmitting the buffered signals according to 
the priority assignments. The transmitted signals occupy the 
RF bandwidth in portions ^ecified by the priority assign- 
ments. The priority assignments can be changed during a 
communication hnk. The subscriber unit receives a trans- 
mission header fi-om another party on the communication 
link, which may include a request by the other party to 
change the priority assignments. If such a request is 
received, the subscriber unit automatically changes the pri- 
ority assignments in response to the request. The commu- 
nication unit is applicable to subscriber units and base 
stations. One such subscriber can be a cellular telephone 
having full-motion video capability. 

21 Claims, 3 Drawing Sheets 
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METHOD AND APPARATUS FOR THE 
DYNAMIC ALLOCATION OF SIGNAL 
BANDWIDTH BETWEEN AUDIO, VIDEO 
AND DATA SIGNALS 

BACKGROUND OF THE INVENTION 

The invention relates in general to wireless communica- 
tion systems and in particular to an RF communication 
system for receiving and transmitting audio, video and data 
signals. 

Today, wireless data solutions are enabling changes of 
great scope and depth in our society. Indeed, the wireless 
information revolution has the potential to democratize the 
information age like never before. Remotely accessible 
computers and data systems are becoming more and more 
available, putting us all on the verge of a world where an 
unlimited amount of information will be available 
anywhere, anytime. 

>^^reless data capabilities are also improving the produc- 
tivity and accessibihty of professionals in the field. The 
ability to send and receive information over airwaves instead 
of copper wires is h*berating the professionals firom their 
ofiSces, giving them immediate access to databases and 
streamlining every aspect of their operations. Already, note- 
book computers equipped with advanced wireless commu- 
nications software and radio frequency modems have 
enabled the formation of "virtual offices," offices that are 
removed from company headquarters. Now, a market ana- 
lysts can track the stock market in his car while sitting in 
traffic during his commute to work. An engineer, instead of 
sitting in his office, can work on a CAD file fi^om the pool 
side of his home. 

The explosion of wireless data communication has been 
fueled by advance in semiconductor technology and soft- 
ware. These advances have allowed audio and data signals 
to be transmitted over digital networks in digital language, 
the language of computers. 

Digital and mixed signal systems offer many advantages 
over old-fashioned analog systems. One important advan- 
tage is the ability of digital systems to transmit and receive 
more information at higher rates. Whereas analog systems 
are limited to transmitting audio at a rate of 64 Kbps, digital 
systems can compress audio transmissions and transmit 
eight times as much information at the same rate. Moreover, 
faster processors have allowed digital systems to transmit 
bits at ever increasing rates. By taking advantage of the 
ability to transmit information more accurately and at higher 
rates, significant savings have been realized in both switch- 
ing capacity and ongoing line costs. 

Additional advantages have been realized through the use 
of multiple access techniques such as Time Division Mul- 
tiple Access ("TDMA") and Code Division Multiple Access 
("CDMA"). These techniques allow for multiple users to 
access a single bandwidth. They also allow for audio and 
data signals transmitted by a single user to be intermingled. 
These techniques make better use of scarce airwave space. 

A recent development in the wireless information revo- 
lution has been the transmission of video signals over the 
airwaves. This is now being done in the television industry, 
as near-perfect images are being transmitted digitally on the 
Ku-band from satellites to home-mounted dishes as small as 
eighteen inches in diameter. A similar development is occur- 
ring in the cellular telephone industry as efforts are being 
made to add video capability to cellular telephones. 

Before quahty video capability can be added to cellular 
telephones, a problem arising from bandwidth limitation 
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must be overcome. Current cellular telephone systems oper- 
ate on a frequency of 900 MHZ. Yet even with the use of 
sophisticated compression routines, the bandwidth is not 
wide enough to transmit the enormous amount of video and 

5 audio information that is required for quality motion pic- 
tures. Bandwidth limitation may not be a problem for high 
frequency satellite transmissions, but it is a problem for the 
comparatively low frequency radio transmissions. 

Therefore, it is an objective of the present invention to 

10 overcome the above-mentioned bandwidth limitation prob- 
lem and provide video communication capability to a radio 
frequency commimication system. 

SUMMARY OF THE INVENTION 

15 

The bandwidth limitation problem is overcome by a 
wireless communication unit comprising at least one digital 
transceiver operable to transmit and receive a plurality of 
data signals over a fixed bandwidth; and a controller oper- 
2Q able to dynamically allocate the fixed bandwidth among the 
plurality of signals. The dynamic allocation is performed by 
making priority assignments to the plurality of signals and 
transmitting the plurality of signals according to the priority 
assignments. 

25 The fixed bandwidth can be an RF bandwidth. The 
plurality of signals can include video, audio and data signals. 

The communication unit can be a subscriber unit or a base 
station. A system empbying the communication units can 
utilize a multiple access technique such as Time Division 

30 Multiple Access or Code Division Multiple Access. One 
embodiment of the communication unit is a video telephone. 

BRIEF DESCRIPTION OF THE DRAWINGS 

no. 1 is a schematic illustration of a cellular communi- 
cation system; 

FIG. 2 is a block diagram of a subscriber imit according 
to the present invention; 

FIG. 3 is a flowchart of steps for the dynamic allocation 
40 of an RF bandwidth among video, audio and data signals, the 
steps being performed by the subscriber unit shown in FIG. 
2; 

FIG. 4 is a block diagram of a transmitter for the sub- 
scriber unit shown in FIG. 2; 

FIG. 5 is a block diagram of a receiver for the subscriber 
unit shown in FIG. 2; and 

FIG. 6 is a block diagram of a base station according to 
the present invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

FIG. 1 shows a cellular communication system 10 includ- 
ing a plurality of subscriber units 12. The subscriber units 12 

55 can include mobile units such as hand held telephones and 
stationary units such as desktop computers. The system 10 
also includes a number of base stations 14 that allow the 
subscriber units 12 to communicate with each other and with 
other communication devices in other networks. 

60 The system 10 covers a geographical area that is divided 
into a grid of cell sites, with each cell site containing at least 
one base station 14. Each base station 14 communicates with 
all subscriber units 12 in its cell site via radio frequency 
("RF") signals. One frequency is used for transmission from 

65 the base station 14 to the subscriber units 12 (the "downlink" 
frequency), and a different frequency is used for transmis- 
sion from the subscriber units 12 to the base station 14 (the 
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"uplink'* frequency). The system 10 employs "frequency 
reuse" to allow more than one base station 14 to operate at 
the same radio frequency. Each cell site is made large 
enough so that RF signals crossing a cell site are attenuated 
in substantial amount so that they are perceived as lower 
level noise by base stations in distant cell sites. Frequency 
isolation occurs in free space because the RF signals are 
inherently attenuated in proportion to the square of the 
distance from the radiating source. Isolation is furthered by 
interference arising from man-made and natural structures. 

One or more frequencies are set aside for setting up a 
communication link or call between the base station 14 and 
a subscriber unit 12. 

The base stations 14 are interlinked with a network 
controller 16 via a distribution facility such as a dedicated 
copper wire or fiber optic network, a radio communication 
link, or a satellite link. The satellite link provides the highest 
system capacity. The network controller 16, in turn, provides 
access to existing wirehne telephone networks 17. Each base 
station 14 determines the received signal strength of each 
call in progress, and forwards this information to the net- 
work controller 16. The network controller 16 uses advanced 
processing technology to keep track of all calls between the 
subscriber units 12 and base stations 14. The network 
controller 16 also uses the signal strength information from 
each base station 14 to determine when a call should be 
"handed ofiE" from the base station in one cell site to the base 
station in another cell site. Hand-off allows communication 
to be maintained with a subscriber unit 12 as the subscriber 
unit 12 roams from cell site to cell site. 

Mdeo, audio and data are transmitted over the airwaves as 
digital signals between the subscriber units 12 and base 
stations 14. Sources of video, audio and data are not limited 
to other subscriber units 12 in the system 10. Since the base 
stations 14 are Hnked to telephone networks, data can be 
provided over wired networks by sources such as private 
faxes and corporate computers containing commercial data- 
bases. Audio can be provided over wired networks by analog 
telephones, personal computers and even radios. Full- 
motion video can be provided by direct broadcast satellites 
and Very Small v^erture Terminals, and by computers over 
fiber optic and ISDN networks. 

Within a cell site, each frequency bandwidth is "shared" 
by all subscriber units 12, either through a lime Division 
Multiple Access ("TDMA**) technique or a Code Division 
Multiple Access ("CDMA*) technique. The TDMA tech- 
nique divides up the total bandwidth into a predetermined 
number of time slots, with each subscriber unit 12 being 
allocated a specific time slot. One of the time slots contains 
an imbedded control channel. Each base station 14 continu- 
ously transmits time division multiplexed bit streams to the 
subscriber units 12 on the downlink frequency, with each 
subscriber unit 12 responding by transmitting bursts on the 
uplink frequency. Even if a base station 14 is not commu- 
nicating with a subscriber unit 12, a dummy time slot 
transmission is sent. 

The CDMA technique, instead of dividing up the total 
bandwidth into time slots, spreads the signal of each sub- 
scriber unit 12 across the entire bandwidth. Although each 
subscriber unit 12 generally occupies the entire bandwidth 
designated by the base station 14, it utilizes only a portion 
of the power available to the base station 14. The 
information-bearing signal is multiplied by a high 
bandwidth, high frequency digital spreading signal, which 
expands the narrow bandwidth information-bearing signal 
into a broad spread signal covering the entire transmission 
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bandwidth. The spreading signal uses quasi-orthogonal bit 
sequences of period Tc, referred to in the art as chips. The 
chip sequence causes the cross-correlation fiinction between 
subscriber units 12 to be small, in which event the subscriber 

5 units 12 are quasi-orthogonal to each other. The chip 
sequence can be generated or chosen so that a predetermined 
or unique chip sequence is assigned to a specific subscriber 
unit 12 each time the subscriber tmit 12 starts a caU. This, of 
course, requires the network controller 16 to maintain a 
central log or listing of all user chip sequence assignments. 

FIG. 2 shows the subscriber unit 12 for the cellular system 
10. The subscriber unit 12 includes a transmitter 18, receiver 
20, controller 22, T/R module 24 and a high eflSciency 
antenna 26. The controller 22 defines and implements the 
protocol for the subscriber unit 12. That is, it defines the 
convention through which the subscriber unit 12 can com- 
municate with the base station 14. The controller 22 decodes 
the header of each base station transmission and executes a 
protocol structure which controls timing and decision mak- 
ing logic instructions (e.g., timing, message slot selection, 

20 T/R control) and other well known operations. Prior to a call 
setup, the subscriber unit 12 monitors air time, activity, 
account numbers, and protocol of the base station 14 to 
determine whether it can access the system 10. When the 
subscriber imit 12 is ready to make a call, or when a call is 

25 transmitted to it, the subscriber unit 12 establishes a setup 
channel with a proximate base station 14. During call setup 
the base station 14 specifies the unique time slots and 
uplink/downlink frequencies for the subscriber unit 12 to 
transmit and receive the call. 

3Q In addition to those well known operations, the controller 
22 allocates the RF bandwidth among the audio, video and 
data signals. \^deo, audio and data signals to be transmitted 
are stored in a video buffer 23a, an audio buffer 23b and a 
data buffer 25c, Each buffer 23a, 23b and 23c is a queue, a 

35 first-in, first-out (FIFO) buffer. The signals are initially 
assigned equal priority ('/s, V3, Vi) of transmission. The 
priority assignments are stored in computer memory such as 
Random Access Memory (RAM) 25. Equal priority assign- 
ments means that all three signals occupy equal portions of 

4Q the RF bandwidth during a transmission by the subscriber 
unit 12. Thus, for each transmission, one third of the RF 
bandwidth is occupied by the video signal, one third of the 
RF bandwidth is occupied by the audio signal, and the 
remaining third of the bandwidth is occupied by the data 

45 signal. The video, audio and data signals could be transmit- 
ted in sequence, or they could be interleaved. 

The priority assigmnents can be changed. If, for example, 
the priority assignments of the video, audio and data signals 
are changed to 1, 0 and 0, the video signal is transmitted over 

50 the entire RF bandwidth tmtil the video buffer 23fl is empty. 
After the video buffer 23a is emptied, the audio and data 
signals are transmitted. If the priority assignments of the 
audio and data signals are equal, each will occupy one-half 
of the RF bandwidth during transmission. If the video buffer 

55 23fl receives additional video, the transmission of the audio 
and data signals is stopped and the video signal is transmit- 
ted until the video buffer 23a is emptied Then, transmission 
of the audio and data signals is resumed. 

If additional data and audio signals must be transmitted 

60 while the video signal is being transmitted, the audio and 
data signals are stored in the audio and video buffers 23^ and 
23c. If the audio and data buffers 23b and 23c become fiill, 
the audio and data already stored in the buffers 236 and 23c 
are pushed out in order to make room for new signals. Thus, 

65 audio and data is lost at the expense of high quahty video. 
The priority assignments and, therefore, bandwidth allo- 
cation arc changed in response to a request from another 
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party on the communication link. If the other party desires 
a higher quality audio, it sends an appropriate request to the 
subscriber unit 12. The controller 22 responds by transmit- 
ting more of the audio signal and buffering more of the video 
and data signals until the fidelity of the transmitted audio 
meets the approval of the other party (i.e., when the other 
parly stops making requests for higher audio fidelity). 

This requires a protocol that is tailored for dynamic 
bandwidth allocation of the video, audio and data signals. To 
implement this protocol, four bits in a transmission header 
are dedicated to the request. Two of the bits indicate whether 
the transmission priority should be increased, decreased, ooi 
changed or reset to preset assignments. The other two bits 
indicate which signal, whether the video signal, audio signal 
or data signal, shotild be affected by an increase or decrease. 
On the transmitting end, the request can be input to the 
controller 22 by means such as a keypad 27. The controller 
22 sets the four bits in the transmission header accordingly. 

On the receiving end, the controller 22 constantly checks 
for requests from the other party by monitoring the trans- 
mission headers received during the communication Unk. 
When the subscriber unit 12 receives a request, its controller 
22 updates the appropriate priority assignments for the audio 
and video signals, llie signals are buffered and the buffers 
23a, 23^ and 23c are emptied in accordance with the priority 
assigrunents. The output of the buffers 23a, 23b and 23c is 
a serial stream, which is supplied to the transmitter 18. 

The transmitter 18 compresses the audio and video signals 
at rates that are initially preset. However, the rates can be 
adjusted by the controller 22 to reduce the amount of 
information being buffered. Following compression, audio, 
video and data signals are formatted, transported and mul- 
tiplexed together with the transmission header (which 
includes the four bit request) to form a composite signal. The 
composite signal is further processed by the transmitter 18 
into either a spread spectrum signal or a time division 
multiplexed signal, depending upon whether CDMA or 
TDMA is being used by the system 10. The encoded signal 
is used to modulate a carrier signal. The modulated carrier 
signal is sent to the antenna 26 through the T/R module 24. 
For subscriber units 12 that do not have a video capability, 
only audio and data signals are mtiltiplexed with the trans- 
mission header to form the composite signal, with compres- 
sion being performed on the audio signal only. 

RF signals received on the antenna 26 are sent to the 
receiver 20 through the T/R module 24. The receiver 20 
separates the incoming signal into four demodulated signals: 
a compressed video signal, a compressed audio signal, a data 
signal and a transmission header. The transmission header is 
sent to the controller 22 and monitored for a request to 
change the priority assignments. The compressed video 
signal, compressed audio signal and the data signal are 
utiformatted. The compressed signals are then decompressed 
using compression rates embedded in the compressed sig- 
nals. The decompressed signals, along with the unformatted 
data signal, are forwarded to the appropriate interfaces 28 in 
the subscriber unit 12. 

The types of interfaces 28 utihzed by the subscriber unit 
12 are partly dependent upon whether the tmit 12 is station- 
ary or mobile. For a mobile subscriber unit 12 such as a 
celhilar telephone, the interfaces 28 must fit within a stan- 
dard cellular phone case. Audio signals sent from the trans- 
milter 18 and received by the receiver 20 can be handled by 
microphones, speakers and their associated circuitry in the 
conventional maimer. Data signals suppHed to the transmit- 
ter 18 and received by the receiver 20 can be transferred in 
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and out of the cellular telephone through a serial or parallel 
port on the case. Video signals received by the cellular 
telephone can be synchronized with the audio signals and 
displayed on a small flat panel display mounted to a surface 

5 of the cellular telephone's case, or they could be supplied to 
a CRT through a parallel port on the case. Video signals 
supplied to the cellular telephone can be provided on a 
parallel or serial port on the case. For example, an automo- 
bile can be equipped with a camera and video capture card 

10 that would supply the video signal to the port. It is even 
possible to furnish the cellular telephone with an internal 
CCD, optical assembly and video processor for providing 
video images directly to the transmitter 18. 

Stationary imits, especially desktop personal computers, 

15 can be equipped with more elaborate interfaces. Audio 
signals can be supplied to a resident soimd card by a hand 
held microphone and they can be outputted from the sound 
card to a speaker system. Data signals can be supplied to the 
transmitter 18 direcdy from computer memory, the comput- 

20 er's motherboard or from communications ports, and data 
from the receiver 20 can be saved in computer memory, 
forwarded to a printer or displayed on a CRT. N^deo signals 
can be supplied by a hand held camera and a resident video 
capture card, with the video image from the camera being 

^ qtiantized by the video capture card in both the spatial 
domain and the intensity domain. Video signals received by 
the computer can be saved in computer memory or displayed 
direcUy on the CRT. 

A desktop computer having a "PENTIUM" processor or 
a more powerful processor can be adapted to operate as a 
subscriber unit 12 with the addition of only a single card. 
The transmitter 18, receiver 20, T/R module 24 and inter- 
faces 28 are mounted to the single card, which is inserted 
into the backplane of the computer. The transmitter 18 and 
receiver 20, which perform analog and digital signal 
processing, are of mixed signal ASIC designs. It is not 
necessary to add a controller 22 to the card; only a Read- 
only Memory (ROM) need be mounted. Instructions for the 
controller are stored in the ROM, and the instructions are 

^ executed by the computer's microprocessor. 

FIG. 3 shows the steps 300-348 that are performed by the 
controller 22 for the dynamic allocation of the RF bandwidth 
among the video, audio and data signals. The controller 22 
monitors each transmission header for a request to change 
priority assignments (step 330). If no request is received 
(step 332), the controller 22 moves the video, audio and data 
from their buffers 23a, 23^ and 23c to the transmitter 18 in 
accordance with the priority assignments (step 334). Thus, if 
the priority assigimients are equal, all three signals are 
equally sent to the transmitter 18. 

If a request is made (step 332), the controller 22 incre- 
mentally changes the priority assignment of the specified 
signal by an increment of, say, 1%, imtil the request has 

55 stopped (step 336). In the meantime, the controller 22 is 
moving the audio, video and data from their buffers 23a, 23b 
and 23c to the transmitter 18 in accordance with the newly 
assigned priorities (step 334). 

The controller 22 also monitors the buffers 23a, 23^ and 

60 23c to detect overflows (step 338). If there is no danger of 
an overflow (steps 340 and 342), the controller 22 continues 
to fill the buffers 23a, 23b and 23c (step 344). If a particular 
buffer 23a, 236 or 23c is in danger of overflowing (step 340), 
the controUer 22 increases the rate at which the transmitter 

65 18 compresses the corresponding signal (step 346). This 
allows the particular buffer 23a, 23b or 23c to be emptied 
faster. Once the danger of overflow has subsided, the com- 



12/02/2003, EAST Version: 1.4.1 



6,111,863 

7 8 

pression rate is reset. If overflow occurs, however, the sent to the T/R module 24, which transmits the signal under 

information in the particular buffer 23a, 23b or 23c must be the control of the controller 22. 

overwritten (step 348). Information stored first in the par- jf the system uses a TDMA technique, the digitally 

ticular buffer 23fl, 23b or 23c is pushed out to make room for encoded information signal is used to modulate a carrier 

new data. 5 frequency only during the allocated time slot (blocks 112 

FIG. 4 shows the functions performed by the transmitter and 114). The resulting burst is transmitted by the T/R 

18. The digital video signal is compressed according to an module. 

algorithm that supports variable rate compression (block pjc. 5 shows the functions performed by the receiver 20. 

102). The digital audio signal is also compressed according x^e receiver 20 performs low noise amplification on the 

to an algorithm that supports variable rate compression 10 signal received from the antenna and T/R module and down 

(block 104). The video and audio signals arc normally at converts the amplified signal into an intermediate frequency 

rates that are preset, subject to change by the controUer 22. pp) signal (block 202). Gain control of the IF signal is 

The compressed video signal is broken up into video performed and the gain-controlled IF signal is mixed to form 

transport packets (block 106), and the audio signal is broken a baseband signal (block 204). The baseband signal is then 

up into audio transport packets (block 108). The data signal, locked onto and demodulated by breaking it into its in-phase 

although uncompressed, is broken up into data transport (I) and quadrature (Q) components, which arc converted into 

packets (block 110). Each transport packet includes a header a digital signal (block 206). The digital signal is deintcr- 

and data portion. In the caise of the compressed signals, the leaved and decoded at a predetermined decoding rate by a 

header will indicate whether the compression rates are convolutional decoder such as a \^terbi algorithm decoder 

stored in the first few bytes of the data portion. '^^ (block 208). The decoded signal is then demultiplexed into 

Apparatus and methods for compressing the audio and a transmission header and video, audio and data transport 

video signals are disclosed in U.S. patent application Ser. packets (block 210). The transmission header is supplied to 

No. 08/580,547, filed Dec. 29, 1995 now U.S. Pat. No. the controller 22 and the audio, data and video signals are 

5,784,572 and incorporated herein by reference. The appa- sent to an inverse transport processor (block 212). The 

ratus supports variable rate compression and utilizes muJ- ^ compressed audio and video signals are iiereafter decom- 

tiple compression algorithms. During a conimunication link, pressed using the audio and video compression rates embed- 

for example, the apparatus can use an MPEG-1 algorithm for ded in the data portion of the transport packet (block 214). 

both audio and video compression. During another commu- The decompressed video and audio signals are synchronized 

nication link, the same apparatus can use an MPEG-2 (if corresponding audio is transmitted) and then forwarded, 

algorithm for video compression and Dolby AC3 for audio along with the unformatted data signal, to the appropriate 

compression. interfaces 28. 

The audio, video and data transport packets are multi- FIG. 6 shows the base station 12 for the cellular system 

plexed together with the transmission header to form a 10. Signals received by an antenna 40 are supplied to a 

composite signal (block 112). The transmission header, receiver module 44 via a T/R module 42. For a CDMA 

which is generated by the controller 22, includes the four system, the receiver module 44 includes a power splitter 

bits that request the other party to change the priorities of the which sends the incoming signal to a plurahty of receivers, 

video, audio and data signals being transmitted by that party. each of which handles a specific communication link. Thus, 

The composite signal is then modulated using phase shift each base station 14 for a CDMA system will employ as 

keying (PSK) modulation, frequency shift keying (FSK) ^ many receivers as communication links it is expected to 

modulation, or any other type of modulation suitable for a establish at a given time. All receivers for the CDMA 

TDMA or CDMA system (block 114). The PSK modulation function in substantially the same manner as the receiver 20 

may be any of binary phase shift keying (BPSK) shown in FIG. 5, except that they employ narrower band 

modulation, quadrature phase shift keying (QPSK) filters and timing loops instead of pilot tracking circuitry, 

modulation, M-ary phase shift keying (MPSK) modulation 45 For a TDMA system, the receiver module 44 contains a 

where M is larger than four, or variants of differential phase single receiver that functions in substantially the same 

shift keying (DPSK) modulation. manner as the receiver shown in FIG. 5, except that it 

FoUowing modulation is forward error correction (block includes a demultiplexer and associated circuitry for for- 

116). Signals to be transmitted are encoded by coding warding the received bursts onto separate communication 

schemes such as Linear Predictive Coding (LPQ or Con- 59 ^^ks. 

tinuously Variable Sloped Delta (CVSD) modulation. Actual Digital audio, data and video signals arc supplied to a 

data bits forming the input signal are interleaved with plurality of transmitters 46, each of which is dedicated to a 

additional bits for ascertaining, or monitoring errors and specific communication link. The modulated carrier signals 

providing for correction. from the various transmitters 46 are combined by a transmit 

If the system uses a CDMA technique, the digitally 55 combiner 48. In the case of the TDMA system, the bursts 
encoded information signal is mixed with a spreading chip from the transmitters 46 are combined at their selected time 
sequence, which is assigned to the subscriber unit (block slots to provide a continuous stream of time-division mul- 
118). The chip sequence is sent by the subscriber unit to the tiplexed information. In the case of the CDMA system, the 
base station 14 as part of the call setup. It is desirable to spread spectrum signals are combined to provide a compos- 
spread the communication signal to cover the entire alio- 60 spread spectrum signal. The combined transmit signal 
cated bandwidth where possible and achieve a high process- frona the transmit combiner 48 is then supplied to the 
ing gain. antenna 40 through the T/R module 42. 

The mixed broad band spread information signal is then The base station 14 is controlled by a cell site controller 

mixed with a carrier frequency to produce the communica- 50 in the same manner that a subscriber unit 12 is controlled 

tion signals to be transmitted (block 120). The specific 65 by its controller 22, except that the cell site controller 50 

frequency used is predetermined by the spectral allocation directs the base station 14 to communicate with all of the 

for the communication system 10. The modulated signal is subscriber units 12 on all of the communication links. The 
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cell site controller 50 also determines signal strength infor- 
mation necessary for a hand off decision , and passes the 
information to the network controller 16. 

The base station 14 also includes an interface 52 for 
sending the video, audio and data signals in digital form to 
the network controller 16. The network controller 16 places 
the audio, video and data signals on a telephone network, 
sends the signals to other base stations, places them on a 
satellite Unk, etc. If the base station 14 has direct access to 
an existing telephone network, the interface 52 would 
include data-to-audio decoders for sending analog audio 
signals over the network and audio-to-data encoders for 
receiving the analog audio signals. 

Thus disclosed is an RF commimication system that 
overcomes the problem of bandwidth limitation associated 
with the transmission of audio and high quality video 
signals. The problem is overcome by dynamically allocating 
the bandwidth among the audio and video signals. 

Further disclosed is a protocol for the transmission of 
audio and video signals. Protocols for current TDMA and 
CDMA systems are not optimized for allocating the RF 
bandwidth during a communication link. 

Still further disclosed is a cellular telephone having 
quality full-motion video capability. 

It is understood that various changes and modifications 
may be made without departing from the spirit and scope of 
the invention. It is also understood that use of the invention 
is not limited to CDMA and TDMA communication 
systems, but can be applied to any other type of narrow 
bandwidth communications system. Accordingly, the 
present invention is not limited to the precise embodiment 
described hereinabove. 

What is claimed is: 

1. A wireless system for the communication of video, 
audio and data signals over an RF bandwidth, comprising: 

a plurality of subscriber imits operable to allocate the RF 
bandwidth among the video, audio and data signals by 
making priority assignments of transmission to the 
video, audio and data signals; 

a pluraUty of base stations covering a geographical area 
divided into a plurality of cell sites, each base station 
being operable to establish RF communications links 
with the subscriber units in its cell site; and 

a network controller connected to the plurality of base 
stations; 

wherein each subscriber unit receives a plurality of trans- 
mission headers during a communication Unk, each 
transmission header including a number of bits indi- 
cating whether a request to change the priority assign- 
ments between the video, audio and data signals has 
been made. 

2. The system of claim 1, wherein the number of bits 
indicate whether a priority assignment of one of the signals 
should be increased, decreased or unchanged. 

3. The system of claim 1, wherein the transmission header 
includes four bits for indicating whether a priority assign- 
ment of one of the signals should be increased, decreased or 
unchanged. 

4. The system of claim 1, wherein at least one subscriber 
unit comprises: 

a digital transceiver operable to transmit and receive a 
data signal and compressed video and audio signals 
over the RF bandwidth; and 

a controller operable to dynamically allocate the RF 
bandwidth among the video, audio and data signals 
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prior to transmission, the dynamic allocation being 
made in response to the transmission header of a 
received signal. 

5. The system of claim 1, wherein at least one base station 
comprises: 

at least one digital transceiver operable to transmit and 
receive data signals and compressed video and audio 
signals over the RF bandwidth; and 

a controller operable to dynamically allocate the RF 
bandwidth among the video, audio and data signals 
prior to transmission, the dynamic allocation allowing 
the video, audio and data signals to be transmitted 
within the RF bandwidth. 

6. A mobile video telephone, comprising: 

a digital transceiver operable to transmit and receive a 
data signal and compressed video and audio signals 
over an RF bandwidth; 

a controller operable to dynamically allocate the RF 
bandwidth among the video, audio and data signals, the 
dynamic allocatioD being performed by making priority 
assignments to each of the video, audio and data signals 
and transmitting the video, audio and data signals 
according to the priority assignments; and 

a display for displaying a video signal received by the 
transceiver; 

wherein the controller performs the steps of: 

buffering the video, audio and data signals to be trans- 
mitted; 

making the priority assignments to each of the buffered 
signals; 

supplying the buffered signals to the transceiver such 
that the buffered signals, when transmitted, occupy 
the fixed bandwidth in portions specified by the 
priority assigtmaents. 

7. The video telephone of claim 6, wherein the transceiver 
includes: 

an antenna; 

a T/R module coupled to the antenna; 

a receiver having an input coupled to the T/R module; and 

a transmitter having an output coupled to the T/R module. 

8. The video telephone of claim 7, wherein the receiver 
includes: 

a demodulator for demodulating an incoming signal from 

the T/R module; 
a demultiplexer for separating the demodulated signal into 

a transmission header and video, audio and data 

signals, the transmission header being supplied to the 

controller; 

a first decompressor for decompressing the video signal 
according to a rate embedded in the video signal; and 

a second decompressor for decompressing the audio sig- 
nal according to a rate embedded in the audio signal. 

9. The video telephone of claim 7, wherein the transmitter 
includes: 

a first compressor for compressing the video signal to fit 
within a video .bandwidth supplied by the controller; 

a second compressor for compressing the audio signal to 
fit within an audio bandwidth supplied by the control- 
ler; 

a combiner for combining the compressed video and 
audio signals; 

a PSK modulator responsive to an output of the combiner; 
and 

a carrier frequency modulator, responsive to an output of 
the PSK modulator. 
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10. The mobile video telephoae of claim 6, wherein the 
controller performs the additional step of changing the rates 
of compression of select signals, whereby the transceiver 
increases the compression of the corresponding signals. 

11. The mobile video telephone of claim 6, wherein the 
controller performs the additional steps of: 

receiving signals including transmission headers from 
another party when a communication link with the 
other party has been established; 

monitoring the transmission headers for requests to 
change the priority assignments; and 

changing the priority assignments in response to the 
requests, whereby the bandwidth are allocated among 
the signals in accordance with the requests. 

12. A wireless communication unit, comprising: 

at least one digital transceiver operable to transmit and 
receive a plurality of signals comprising at least two 
from the group consisting of video signals, audio 
signals, and data signals; 

a controller operable to dynamically allocated the fixed 
bandwidth among the plurality of signals, the dynamic 
aUocation being performed by making priority assign- 
ments to eadi of the plurality of signals and transmit- 
ting the plurality of signals according to the priority 
assignments, wherein said controller supplies buffered 
signals to the transceiver such that the buffered signals, 
when transmitted, occupy the fixed bandwidth in por- 
tions specified by the priority assignments; and 

a plurality of queues corresponding to the plurality of 
buffers, the controller buffering the plurality of signals 
in the queues; 

wherein the controller performs the steps of: 
buffering the signals to be transmitted; 
making tbe priority assigiunents to each of the buffered 
signals; 

receiving signals including transmission headers from 
another party when a communication link with the 
other party has been established; 

monitoring the transmission headers for requests to 
change the priority assignments; and 

changing the priority assignments in response to the 
requests, whereby the fixed bandwidth is allocated 
among the plurality of signals in accordance with the 
requests. 

13. The communication unit of claim 12, wherein the 
priority assignments are changed incrementally until the 
requests have stopped. 

14. The communication unit of claim 13, wherein each 
request occupies four bits in the transmission header. 

15. A wireless communication unit, comprising: 

at least one digital transceiver operable to transmit and 
receive a plurality of signals over a fixed bandwidth; 
and 

a controller operable to dynamically allocate the fixed 
bandwidth among the plurality of signals, the dynamic 
allocation being performed by making priority assign- 
ments to each of the plurality of signals and transmit- 
ting the plurality of signals according to the priority 
assignments wherein the commimication unit is a sub- 
scriber unit; 

wherein the transceiver includes: 
an antenna; 

a T/R module coupled to the antenna; 
a receiver having an input coupled to the T/R module; 
and 
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a transmitter having an output coupled to the T/R 

module; and 
wherein the receiver further includes: 

a demodulator for demodulating an incoming signal 

from the T/R module; 
a demultiplexer for separating the demodulated sig- 
nal into a transmission header and the plurality of 
signals, the transmission header being supplied to 
the controller, and 
at least one decompressor for decompressing select 
signals of the plurality according to rates embed- 
ded in the select signals. 

16. A wireless communication unit, comprising: 
at least one digital transceiver operable to transmit and 

receive a plurality of signals over a fixed bandwidth; 
and 

a controller operable to dynamically allocated the fixed 
bandwidth among tbe plurality of signals, the dynamic 
allocation being performed by making priority assign- 
ments to each of the plurality of signals and transmit- 
ting the plurality of signals according to the priority 
assignments wherein the commimication unit is a sub- 
scriber unit; 

wherein the transceiver includes: 
an antenna; 

^ a T/R module coupled to the antenna; 

a receiver having an input coupled to the T/R module; 
and 

a transmitter having an output coupled to the T/R 

module; and 
wherein the transmitter further includes: 

at least one compressor for compressing select sig- 
nals of the plurality at compression rates supplied 
by the conU'oUer; 
a combiner for combining the plurality of signals; 
a PSK modulator re^onsive to an output of the 

combiner; and 
a carrier frequency modulator, responsive to an out- 
put of the PSK modulator. 

17. The communication unit of claim 16, wherein the 
transmitter further includes an encoder between the com- 
biner and PSK modulator for generating a code division 
spread ^eclrum signal from an output of the combiner, the 
spread ^ectrum signal being supplied to the PSK modula- 
tor. 

18. The communication unit of claim 16, wherein the 
transmitter further includes an encoder between the com- 
biner and PSK modulator for generating a time division 
multiplexed signal from an output of the combiner, the time 
division multiplexed signal being supplied to the PSK 
modulator. 

19. A method of communicating a plurality of signals 
within an RF bandwidth, comprising the steps of: 

making priority assignments to each of the plurality of 
signals; 

transmitting the plurality of signals within the RF band- 
width according to the priority assignments, wherein 
the transmitted signals occupy tbe RF bandwidth in 
portions specified by the priority assignments; 
gQ buffering the signals prior to the step of making the 
priority assignments, wherein the signals are buffered 
on a first-in, first-out basis; 

establishing a communication link with another party; 

receiving signals including transmission headers from the 
65 other party; 

monitoring the transmission headers for requests to 
change the priority assignments; and 
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changing the priority assignments in response to the 
requests, whereby the RF bandwidth is allocated 
among the plurality of signals in accordance with the 
requests. 

20. The method of claim 19, wherein the priority assign- 
ments arc changed incrementally until the requests have 
stopped. 

21. A method of communicating a plurality of signals 
within an RF bandwidth, comprising the steps of: 

making priority assignments to each of the plurality of 
signals; 

transmitting the plurality of signals within the RF band- 
width according to the priority assignments, wherein 
the transmitted signals occupy the RF bandwidth in 
portions specified by the priority assignments; 
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buffering the signals prior to the step of making the 
priority assignments, wherein the signals are buffered 
on a first-in, first-out basis; 

receiving a transmission; 

demodulating the transmission; 

separating the transmission into video, audio and data 
signal; 

decompressing the video signal according to a rate 
embedded in the video signal; and 

decompressing the audio signal according to a rate 
embedded in the audio signal. 
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